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(57) Abstract: A system for fusing images comprises sensors (102 and 104) for generating sets of image data. An information 
processor (108) receives and samples the sets of image data to generate sample data for computing a fused image array. A display 
(110) receives the fused image array and displays a generated fused image. Step 1 of 4 for fusing images receives sets of image data 
generated by sensors (102 and 104). Step 2 samples the sets of image data to produce sample data. Step 3 computes a fused image 
array from the sample data. Step 4 displays a fused image generated from the fused image array on a display (110). Step I of 4 for 
computing a fused image array samples sets of image data generated from sensors (102 and 104) to produce sample data. Image 
fusion metrics from the sample data are determined in step 2. Step 3 calculates weighting factors from the image fusion metrics. 
Step 4 computes a fused image array from the weighting factors, wherein the fused image array is used to generate the fused image. 
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METHOD AND SYSTEM FOR FUSING IMAGES 



TECHNICAL FTRLD OF THE INVENTION 

This invention relates generally to the field of electro-optical systems and 
more specifically to a method and system for fusing images. 



5 



0184e28A1 I > 



wo 01/84828 



PCTAJSOl/12260 



PACKOROUND OF THE INVENTION 

Image fusion involves combining two or more images produced by two or 
more image sensors into one single image. Producing one image that mitigates the 
weak aspects of the individual images while retaining the strong ones is a complicated 
5 task, often requiring a mainframe computer. Known approaches to image ftision have 

not been able to produce a small, lightweight system that consumes minimal power. 

Known approaches to fusing images require a great deal of computing power. 
To illustrate, suppose that two image sensors each have the same pixel arrangement. 
Let Nh be the number of horizontal pixels, and let A^v be the number of vertical pixels, 
1 0 such that the total number of pixels per sensor is Nh A^v. Let the frame rate of the 

display be Q^, expressed in Hz. The time allowed for processing each frame is 
given by: 

l/Qd 

All processing for each displayed image must be done vsdthin this time to keep the 
15 system operating in real time. To calculate the processing time per pixel, first 

compute the total number of pixels of both sensors. Given that each image sensor has 
the same pixel arrangement, the total number of pixels for both sensors is: 

The processing time 2^ per pixel is given by: 
20 r,= -i^ = ^ 

The processing time Tp per pixel is the maximum amount of time allotted per pixel to 
calculate a display pixel from the two corresponding sensor pixels, while allowing for 
real time processing. For example, given an average system where = 30 Hz, N/, = 
640, and Nv = 480, the processing time r^per pixel is: 
25 rp= 108.5 ns 

For handheld or portable applications, processor speed Qp is limited to about 
150 MHz. A maximum of approximately two instructions per cycle is allowed in 
current microprocessors and digital signal processors. The time required per 
instruction is given by: 
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= 3.33 ns 



The number of instruction cycles allowed for each pixel in real time processing is 
given by: 

r 

M = — = 32 instruction cycles 
^, 

5 Thirty-two instruction cycles per pixel is often not a sufficient number of cycles, 

especially considering the fact that a simple "divide, floating point" , could easily 
require 10 to 100 instruction cycles to complete. Practical image fusion systems 
generally require over 100 instruction cycles per pixel, and sophisticated image fusion 
algorithms often require over 1,000 instruction cycles per pixel. Consequently, 
1 0 current image fusion systems are confined to mainfi*ame computers. 

While known approaches have not been applied to handheld or portable 
applications, the challenges in the field of image fiision have continued to increase 
with demands for small, lightweight systems that consume minimal power. 
Therefore, a need has arisen for a new method and system for fusing images. 

15 
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SUMMARY OF THE INVENTION 

In accordance with the present invention, a method and system for fusing 
images are provided that substantially eliminate or reduce disadvantages, and 
problems associated with previously developed systems and methods. 
5 A system for fusing images is disclosed. The system comprises two or more 

sensors for generating two or more sets of image data. An information processor 
receives and samples the sets of image data to generate sample data and computes a 
fused image array from the sample data. A display receives the fused image array and 
displays a fused image generated from the fused image array. 

10 A four-step method for fusing images is disclosed. Step one calls for 

receiving sets of image data generated by sensors. Step two provides for sampling the 
sets of image data to produce sample data. In step three, the method provides for 
computing a fused image array from the sample data. The last step calls for 
displaying a fused image generated from the fused image array. 

1 5 A four-step method for computing a fused image array is disclosed. Step one 

calls for sampling sets of image data generated from sensors to produce sample data. 
Step two provides for determining image fusion metrics from the sample data. Step 
three calls for calculating weighting factors from the image fusion metrics. The last 
step provides for computing a fused image array from the weighting factors, wherein 

20 the fused image array is used to generate the fused image. 

A technical advantage of the present invention is that it computes the fused 
image from sampled sensor data. By sampling the sensor data, the invention reduces 
the number of instruction cycles required to compute a fused image. Reducing the 
number of instruction cycles allows for smaller, lightweight image fusion systems that 

25 consume minimal power. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present invention and for further 
features and advantages, reference is now made to the following description, taken in 
conjunction with the accompanying drawings, in which: 
5 FIGURE 1 is a system block diagram of one embodiment of the present 

invention; 

FIGURE 2 is a flowchart demonstrating one method of fusing images in 
accordance with the present invention; 

FIGURE 3A illustrates sampling with a fixed array pattern in accordance with 
10 the present invention; 

FIGURE 3B illustrateis sampling with a varied array pattern in accordance 
with the present invention; 

FIGURE 3C illustrates sampling randomly in accordance with the present 
invention; and 

15 FIGURE 4 illustrates a method of computing weighting factors in accordance 

with the present invention. 
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DETAILED DESCRIPTION OF THE DRAWINGS 

FIGURE r is a system block diagram of one embodiment of the present 
invention. In this embodiment, a sensor A 102 and a sensor B 104 detect one or more 
physical objects 106 in order to generate image data to send to an information 
5 processor 108, which fuses the sets of image data to produce a single fused image to 

be displayed by a display 1 10. The sensor ^4 102 detects the physical objects 106 and 
generates sensor data, which is sent to an amplifier y4 1 12. Amplifier^ 1 12 amplifies 
the sensor data and then sends it to an analog-to-digital convertery4 114. The analog- 
to-digital converter A 114 converts the analog sensor data to digital data, and sends 

1 0 the data to an image buffer A 1 16 to store the data. The sensor B operates in a similar 

fashion. The sensor 5 104 detects the physical objects 106 and sends the data to 
amplifier 5 118. The amplifier 5 118 sends amplified data to an analog-to-digital 
converter B 120, which sends converted data to an image buffer B 122. A field 
programmable gate array 124 receives the data generated by the sensor^ 102 and the 

15 sensor B 104. The information processor 108 receives the data from the field 

programmable gate array 124. The information processor 108 generates a fused 
image from the sets of sensor data, and uses an information processor buffer 126 to 
store data while generating the fused image. The information processor 1 08 sends the 
fused image data to a display buffer 128, which stores the data imtil it is to be 

20 displayed on the display 1 1 0. 

FIGURE 2 is a flowchart demonstrating one method of image fusion in 
accordance with the present invention. The following steps may be performed 
automatically using an information processor 108. The method begins with step 202, 
where two or more image sensors generate two or more sets of image data. As above, 

25 suppose that there are two image sensors, each with the same pixel arrangement. Let 

Nf, be the number of horizontal pixels, and let Ny be the number of vertical pixels, 
such that the total number of pixels per sensor is Nh • Ny, The sensors may 
comprise, for example, visible light or infrared light image detectors. Assume that 
detectable variations in the proportion of the fused image computed from one set of 

30 image data and from the other set of image data occiu* in time r^, where: 
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Hence, the computation of the proportion does not need to be calculated at each 
frame. Also, assume that the information required to form a metric that adjusts the 
system to a given wavelength X proportion can be derived from fewer than N^^Ny 
pixels. 

The method then proceeds to step 204 where the sets of image data are 
sampled to produce sample data. FIGURES 3A, 3B, and 3C illustrate three methods 
of sampling image data in accordance with the present invention. FIGURE 3A 
illustrates sampling with a fixed array pattern. The sampled pixels (/, J) 302, 304, 
306, 308, 310, 312, 314, 316, and 318 may be described by: 



One possible arrangement is to have A/, = 2 for the horizontal difference between one 
sampled pixel to the next sampled pixel, and Ay = 2 for the vertical difference 
between one sampled pixel to the next sampled pixel. The groups of pixels 320 and 
322, each with 2 sampled pixels 302 and 304, and 308 and 310, respectively, are 
sampling blocks. FIGURE 3B illustrates sampling with a varied array pattern. 
FIGURE 3C illustrates random sampling. A sequence of sampling patterns may also 
be used, repeating at any given number of sampling cycles, or never repeating, as in a 
random pattern for each continued sampling cycle. 

Referring again to FIGURE 2, in steps 206 to 210, a fused image array is 
computed from the sample data. In step 206, image fiision metrics are calculated 
from the sample data. The image fusion metrics are values assigned to the pixels of 
the sample data. These values, for example, may give the relative weight of the data 
from each sensor, such that the data from the sensor that produces the better image is 
given more weight. Or, these values may be used to provide a control for the 
production of, for example, a false color image. All the pixels may be assigned the 
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same metric, p, or each sample pixel may assigned its own metric, p,y, where the 
subscript ij designates the pixel in the /th row and yth column. 

In step 208, weighting factors a^, where the subscript ij designates the pixel in 
the rth row and yth column, are calculated from the image fusion metrics. The 
5 weighting factors are values assigned to the pixels of the fused image. The weighting 

factors may be computed by, for example, linear interpolation of the image fusion 
metrics. 

FIGURE 4 illustrates a method of computing weighting factors in accordance 
with the present invention. For example, suppose that the sample data was sampled 
10 using a fixed array pattern, where every fifth point 402, 404, 406, and 408 is sampled 

in both the horizontal and vertical direction, that is. A/, = Av = A = 5. A sampling 
block 410 comprises to two sampled points 402 and 404. The weighting factors a,j of 
the first row may be computed in the following manner. First, an incremental value 
for the first row in the horizontal direction 6^ is calculated using the following 
1 5 formula: 

6/„=(Pi6-pn)/A. 

Then, the weighting factors between Ph and Pi6 in the horizontal direction are 
calculated using the following formula: 
ajy=Pii +bhx (/-I) 

20 The weighting factors in the vertical direction between pu and Pei are calculated in a 

similar manner, using the following equations: 
6vi=(P6!-pii)/A 
a/i = Pii +6vi (/- 1) 

Referring again to FIGURE 2, the method then proceeds to step 210, where a 
25 fused image array, which is used to generate a fused image, is computed from the 

weighting factors. An array of weighting factors a,y generates the following fused 
image array: 

where / e {1, Nh}J 6 {1, iVy}, the superscripts {d) denotes display, {A) denotes 
30 sensor and {B) denotes sensor 5, and Vfj corresponds to the voltage at pixel {ij). 
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The fused image array describes the relative weights of the data from each sensor. 
Weighting factor gives the relative weight of the voltage from sensor A at pixel 
weighting factor (l-a,y) gives the relative weight of the voltage from sensor B at 
pixel (ij). This example shows a linear weight; other schemes, however, can be used. 
The method then proceeds to step 212, where the fused image generated from the 
fused image array is displayed on a display 1 1 0. 

By sampling the image data, this embodiment allows for more instruction 
cycles to calculate P,y for each sampled pixel. To calculate the number of instruction 
cycles available for each sampled pixel, first calculate the total number of instruction 
cycles per sampled pixel, and then subtract number of cycles per pixel needed to 
sample the pixels and to compute the fused image metrics and the fused image array. 
For example, assume that data is sampled using fixed array sampling. The total 
number of instructions for each sampled pixel is given by: 

where Xs is the processing time per sampled pixel, which is given by: 
1 

where rih and «v are the number of sampled pixels in the horizontal direction and in 
the vertical direction, respectively. Sampling each sampling block, without double 
counting borders, requires about (A + 1)[2(A - 1) + 6] instruction cycles. Each 
sampling block contains two sampled pixels, so each sampled pixel loses 1/2(A + 
1)[2(A - 1) + 6] instruction cycles per pixel. Computing the fused image array from 
the weighting factors requires approximately four instruction cycles for each 
calculation, that is: 

Therefore, the time left per pixel for calculating the image fusion metrics Pj, is: 

r AN N 

N,= ^- 1/2(A+ l)[2(A-l) + 6]- ^ 

r, 
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Using the values given above: = 30 Hz, N,, = 640, = 480, A/, = Av = 5, n,, = 128, 
and Wv = 96, the number of instruction cycles is computed to be: 

Nj = 269 instruction cycles. 
This is a dramatic improvement compared with the 32 cycles allotted in conventional 
5 methods. The extra cycles may be used for more complex calculations of fiy or other 

features. Moreover, if p,y is assumed to be the same for all pixels, even more 
additional cycles may be available to determine p,y, allowing for a more sophisticated 
manipulation. 

Although an embodiment of the invention and its advantages are described in 
10 detail, a person skilled in the art could make various alternations, additions, and 

omissions without departing from the spirit and scope of the present invention as 
defined by the appended claims. 
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WHAT I S C LA I MEP IS : 

1 . A system for fusing images, the system comprising: 

a. two or more sensors for generating two or more sets of image data; 

b. an information processor for receiving and sampling the sets of image 
5 data to generate sample data and for computing a fused image array from the sample 

data; and 

c. a display for receiving the fused image array and displaying a fused 
image generated from the fused image array. 

10 2. The system of Claim 1 further comprising a field programmable gate array 

coupled to the sensors and the information processor. 

3. The system of Claim 1 further comprising one or more converters operable to 
convert analog signals from the sensors to digital data for use by the information 
1 5 processor. 



4. The system of Claim 1 wherein the fiised image array assigns a value to one or 
more pixels of the fused image, wherein the value describes the relative weights of the 
sets of image data. 

20 
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5. A method for fusing images, the method comprising: 

a. receiving two or more sets of image data generated by two or more 
sensors; 

b. sampling the sets of image data to produce sample data; 

c. computing a fiised image array from the sample data; and 

d. displaying a fused image generated from the fused image array. 



6. The method of Claim 5, wherein the sampling step further comprises sampling 
with a fixed array pattem. 

10 

7. The method of Claim 5, wherein the sampling step further comprises ssimpling 
with a varied array pattem. 

8. The method of Claim 5, wherein the sampling step further comprises sampling 
15 randomly. 

9. The method of Claim 5, wherein the computing step further comprises 
determining one or more image fusion metrics, wherein the image fusion metrics are 
values assigned to one or more pixels of the sample data. 

20 

1 0. The method of Claim 5, wherein the computing step further comprises 
calculating one or more weighting factors from the image fusion metrics, wherein the 
weighting factors are values assigned to one or more pixels of the fused image. 

25 11. The method of Claim 10, wherein the computing step further comprises 

calculating the weighting factors by interpolation of the image fiision metrics. 

12. The method of Claim 10, wherein the computing step further comprises 
calculating the weighting factors by linear interpolation of the image fusion metrics. 

30 
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13. The method of Claim 5, further comprising performing the foregoing steps 
automatically using an information processor. 
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14. A method for computing a fused image array, the method comprising: 

a. sampling the sets of image data generated from two or more sensors to 
produce sample data; 

b. determining one or more image fusion metrics from the sample data, 
5 wherein the image fusion metrics are values assigned to one or more pixels of the 

sample data; 

c. calculating one or more weighting factors from the image fusion 
metrics, wherein the weighting factors are values assigned to one or more pixels of a 
fused image; and 

1 0 d. computing a fused image array from the weighting factors, wherein the 

fused image array is used to generate the fused image. 

15. The method of Claim 14, wherein the fused image array describes the relative 
weights of the sets of image data at one or more pixels of the fused image. 

15 

16. The method of Claim 14, wherein the sampling step further comprises 
sampling with a fixed array pattern. 

1 7. The method of Claim 14, wherein the sampling step further comprises 
20 sampling with a varied array pattern. 

18. The method of Claim 14, wherein the sampling step further comprises 
sampling randomly. 

25 1 9. The method of Claim 14, wherein the calculating step fiirther comprises 

calculating the weighting factors by interpolation of the image fusion metrics. 

20. The method of Claim 14, wherein the calculating step further comprises 
calculating the weighting factors by linear interpolation of the image fusion metrics. 

30 
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21 . The method of Claim 14, further comprising perfomiing the foregoing steps 
automatically using an information processor. 
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